Skip to content

refactor(stm): cache the circuit test fixtures on disk - #3481

Merged
hjeljeli32 merged 9 commits into
mainfrom
hjeljeli32/3462-extend-on-disk-cache-recursive-circuit-fixtures
Aug 20, 2026
Merged

refactor(stm): cache the circuit test fixtures on disk#3481
hjeljeli32 merged 9 commits into
mainfrom
hjeljeli32/3462-extend-on-disk-cache-recursive-circuit-fixtures

Conversation

@hjeljeli32

@hjeljeli32 hjeljeli32 commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator

Content

This PR extends the content-keyed FileMutex cache introduced by #3433 to both circuit test trees, which had none. Under the nextest process-per-test model an in-process cache amortizes nothing, so the SRS, the derived recursive verifying key, the certificate circuit keys and the deterministic 3000-signer fixture are now computed once per fingerprint and shared across test processes and runs.

Measured locally: circuits::halo2_ivc slow tier 435 s → 381 s, circuits::halo2 slow tier 50 s → 44 s (−12 % each, ±8 s run to run). Numbers per ticket item are in #3462.

Changes

  • Shared SRS (halo2_ivc/tests/common/generators/setup.rs): build_shared_recursive_context reads the degree-19 SRS from the same "unsafe-srs" entry IvcSnarkProverSetup::build_for_test writes, both deriving from the same seed. A compile-time assertion ties ASSET_SEED to UNSAFE_SRS_SEED, so a change to either cannot silently alter the committed assets.
  • Recursive verifying key: cached as a single blob under a fingerprint that folds in the freshly derived certificate verifying key, the committed production recursive key, both circuit degrees and the SRS seed. The certificate key is derived on every call (~93 ms) because its bytes are part of the cache address.
  • Certificate circuit keys (halo2/tests/golden/helpers.rs): the process-local LazyLock map is replaced by the on-disk KeyProvider, behind the cache lock so parallel processes racing a cold miss derive the pair once. The SRS asset filename now carries the seed that produced it, so a seed change cannot pair an old SRS with keys fingerprinted for a new one.
  • Signer fixture: the generator-derived values are cached as CBOR; everything else in AssetGenerationSetup is a pure function of them and is recomputed on both paths, so the two cannot disagree. Validated on load — vector lengths, plus the genesis signature verified against the message derived from the cached tree, which transitively covers tampering with the tree, the key or the signature.
  • Cache contract: entries are rebuilt on an absent, unreadable, undecodable, non-canonical or validator-rejected file, and never surface as an error. It makes no wider claim: canonical encoding shows the bytes are self-consistent, not who wrote them.
  • Tests: 11 new, all fast and hermetic — cold miss, warm hit, truncation, trailing bytes, validator rejection, the real signer-fixture validator (fresh / wrong lengths / changed tree / changed key or signature), and input-by-input coverage of all three production cache fingerprints.

Pre-submit checklist

  • Branch
    • Tests are provided (if possible)
    • Crates versions are updated (if relevant)
    • CHANGELOG file is updated (if relevant)
    • Commit sequence broadly makes sense
    • Key commits have useful messages
  • PR
    • All check jobs of the CI have succeeded
    • Self-reviewed the diff
    • Useful pull request description
    • Reviewer requested
  • Documentation
    • No new TODOs introduced

Issue(s)

Closes #3462

@hjeljeli32 hjeljeli32 self-assigned this Aug 17, 2026
@hjeljeli32 hjeljeli32 added the run-slow-tests Technical label to run slow tests tiers in the CI. label Aug 17, 2026
@github-actions

github-actions Bot commented Aug 17, 2026

Copy link
Copy Markdown

Test Results

     5 files  ± 0     209 suites  ±0   2h 12m 44s ⏱️ + 1h 33m 21s
 3 462 tests +70   3 462 ✅ +70  0 💤 ±0  0 ❌ ±0 
11 325 runs  +73  11 325 ✅ +73  0 💤 ±0  0 ❌ ±0 

Results for commit 27ce251. ± Comparison against base commit e5f094c.

♻️ This comment has been updated with latest results.

@hjeljeli32
hjeljeli32 deployed to testing-preview August 17, 2026 02:35 — with GitHub Actions Active
@hjeljeli32
hjeljeli32 deployed to testing-2-preview August 17, 2026 02:35 — with GitHub Actions Active

@damrobi damrobi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some none blocking comments but otherwise it looks good 👍

Comment thread mithril-stm/src/circuits/halo2_ivc/tests/common/generators/setup.rs Outdated
Comment thread mithril-stm/src/circuits/halo2_ivc/tests/common/generators/setup.rs
Comment thread mithril-stm/src/circuits/halo2_ivc/tests/common/generators/setup.rs Outdated
Comment thread mithril-stm/src/circuits/halo2_ivc/tests/common/generators/setup.rs Outdated
Comment thread mithril-stm/src/circuits/halo2_ivc/tests/common/generators/setup.rs Outdated
Comment thread mithril-stm/src/circuits/halo2_ivc/tests/common/generators/setup.rs Outdated
Comment thread mithril-stm/src/circuits/halo2_ivc/tests/common/generators/setup.rs Outdated
Comment thread mithril-stm/src/circuits/halo2/tests/golden/helpers.rs Outdated
Comment thread mithril-stm/src/circuits/halo2/tests/golden/helpers.rs Outdated
Comment thread mithril-stm/src/circuits/halo2_ivc/tests/common/generators/setup.rs
@hjeljeli32
hjeljeli32 force-pushed the hjeljeli32/3462-extend-on-disk-cache-recursive-circuit-fixtures branch from 500c53b to 27ce251 Compare August 20, 2026 04:07
@hjeljeli32
hjeljeli32 requested a review from damrobi August 20, 2026 04:08
@hjeljeli32
hjeljeli32 deployed to testing-2-preview August 20, 2026 05:16 — with GitHub Actions Active
@hjeljeli32
hjeljeli32 deployed to testing-preview August 20, 2026 05:16 — with GitHub Actions Active

@damrobi damrobi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

@hjeljeli32
hjeljeli32 merged commit c8ff793 into main Aug 20, 2026
50 checks passed
@hjeljeli32
hjeljeli32 deleted the hjeljeli32/3462-extend-on-disk-cache-recursive-circuit-fixtures branch August 20, 2026 08:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run-slow-tests Technical label to run slow tests tiers in the CI.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Extend the on-disk test cache to the recursive circuit test fixtures

2 participants